Image Pickup Processing Method and Image Pickup Apparatus

ABSTRACT

A roll correction section ( 21 ) performs correction to an image captured by a camera ( 11 ). A region limiting section ( 15 ) limits image data of an image in an upper region of the image and outputs the image data and a roll amount detection section ( 13 ) detects, from the limited and output image data, a roll amount of the image. A roll correction section ( 21 ) performs correction based on the detected roll amount.

TECHNICAL FIELD

The present invention relates to an image processing technique forcapturing an image by a camera and performing correction, based onfeature data detected from a captured image.

BACKGROUND ART

Conventionally, techniques for eliminating the inclination of a camerahave been proposed. In such techniques, an inclination of a camera isreduced by extracting edges in an image, and then, assuming that theinclination of an edge having the largest length is the inclination ofthe camera, affine transform is performed to image data to eliminate theinclination of the edge having the largest length (e.g., see PatentReference 1).

Moreover, a technique for extracting a vertical edge in an image anddetecting an inclination of the image from the vertical edge isdisclosed in Non-patent Reference 1.

(Patent Reference 1) Japanese Laid-Open Publication No. 4-314274 (Page2, FIG. 1)

(Non-patent Reference 1) Hidekazu Ohmi and Fumihiko Saitoh, “ImageInclination Measurement Based on Edge Directions and an Analysis ofRelation between Visual Inclination”, MVA2002 IAPR Workshop on MachineVision Application, Dec. 11-13, 2002.

PROBLEMS THAT THE INVENTION IS TO SOLVE

In image capturing, an image is normally captured in illumination fromabove. Therefore, the ground, the floor or the like often comes out in alower portion of the image. If processing such as roll correction andbacklight correction is performed to such an image based on feature dataof the entire image, desired correction effects can not be achieved inmany cases.

For example, in roll correction of an image for making perpendicularedges of an artificial object be vertical in the image, when detectionof edges is performed in the entire image, the floor or the ground comesout under the object and thus roll correction is influenced by edgescontained in the image of the floor or the ground. Therefore, rollcorrection might not be properly performed.

Moreover, in exposure correction such as backlight correction, it ishighly possible that intensity distribution is largely different betweenan upper portion and a lower portion of the image. Therefore, whenfeature data of the entire image is used, there may be cases wheredesired correction effects can not be achieved.

In view of the above-described problems, it is therefore an object ofthe present invention to make it possible to execute correction of animage captured by a camera in a more proper manner than in the knowntechniques.

DISCLOSURE OF INVENTION

The present invention is directed to obtaining feature data for use incorrection of an image not from an entire image but from an upperportion of the image. The present invention has been devised inconsideration that in a normal captured image, an image capturing statesuch as characteristics of an imaged subject is different between upperand lower portions of the image. For example, in roll correction using aperpendicular edge, a region in which a perpendicular edge element isobtained is limited to an upper portion of an image, thereby allowingelimination of influences of an edge in the floor or the ground.Moreover, in exposure correction such as backlight correction, a regionin which an intensity value is obtained is limited to an upper portionof an image, thereby allowing elimination of influences of a regionwhich is hardly exposed to illumination from above.

That is, the present invention provides, as an image capturingprocessing method, a method in which an image is captured by a camera,feature data of the image is detected in an upper region of the capturedimage, a correction amount is calculated from the feature data, andthen, based on the calculated correction amount, correction control ofan optical system of the camera or correction of the image data isperformed.

Moreover, the present invention provides, as an image capturing system,a system including a camera for capturing an image, a region limitingsection for limiting image data of an image captured by the camera to anupper region of the image and outputting the limited image data, acorrection amount calculation section for detecting, from the limitedimage data output from the region limiting section, feature data of theimage and calculating a correction amount from the feature data, and acorrection section for performing, based on a correction amountcalculated by the correction amount calculation section, correctioncontrol of an optical system of the camera or correction of the image.

According to the present invention, a correction amount is calculatedbased on feature data detected not from an entire captured image butfrom an upper region of the image, and correction control of an opticalsystem of a camera or correction of image data is performed based on thecalculated correction amount. Thus, information to be noise is removedin advance during calculation of a correction amount, so that a moreproper correction amount can be calculated. Accordingly, correction ofan inclination and a motion of an image can be executed more accurately.

In this case, as a simple example, the upper region means to beapproximately upper half of the entire region of an image. Moreover, theupper region may be a rectangular or oval region contained in theapproximately upper half of the entire region. Alternatively, the upperregion may be a region of which barycentric position is located higherthan a predetermined threshold (e.g., a center position of the entireimage) among regions obtained by dividing the image by a knownsegmentation method.

Moreover, in exposure control, an intensity value may be taken intoconsideration when an upper region setting is made. For example, aregion in which an intensity value is a predetermined threshold or morein the above-described upper region may be set as an upper region, and aregion of which barycentric position is located higher than apredetermined threshold (e.g., a center position of the entire image) inthe region in which an intensity value is a predetermined threshold ormore may be an upper region.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating the configuration of an imagecapturing system according to a first embodiment of the presentinvention.

FIG. 2 is a flowchart illustrating the operation of the image capturingsystem of FIG. 1.

FIG. 3( a) is a view illustrating an image before correction; FIG. 3( b)is a view illustrating vertical edges extracted from the image of FIG.3( a); and FIG. 3( c) is a view illustrating an image after correction.

FIG. 4 is a graph showing image data in the xy coordinate system.

FIG. 5 is a block diagram illustrating the configuration of an imagecapturing system, in which region limitation is performed, according tothe first embodiment of the present invention.

FIG. 6 is a block diagram illustrating the configuration of an imagecapturing system according to a second embodiment of the presentinvention.

FIG. 7 is a flowchart illustrating the operation of the image capturingsystem of FIG. 6.

FIG. 8 is a block diagram illustrating the configuration of an imagecapturing system, in which region limitation is performed, according tothe second embodiment of the present invention.

FIG. 9 is a block diagram illustrating the configuration of an imagecapturing system according to a third embodiment of the presentinvention.

FIG. 10 is a block diagram illustrating the configuration of an imagecapturing system according to the third embodiment of the presentinvention.

FIG. 11 is a block diagram illustrating an exemplary configurationaccording to a modified example of the third embodiment of the presentinvention.

FIG. 12 is an illustration of an exemplary application form of themodified example of the third embodiment of the present invention.

FIG. 13 is a block diagram illustrating the configuration of an imagecapturing system according to a fourth embodiment of the presentinvention.

FIG. 14 shows graphs schematically illustrating illumination frequencydistributions in a proper illumination state and in a backlight state,respectively.

FIG. 15 is a block diagram illustrating another exemplary configurationof the image capturing system according to the fourth embodiment of thepresent invention.

FIG. 16 is a block diagram illustrating an exemplary configuration of animage capturing system according to a fifth embodiment of the presentinvention.

FIG. 17 is an illustration of an exemplary application form of the fifthembodiment of the present invention.

FIG. 18 shows graphs conceptually illustrating synthesis of rollcorrection amounts according to the fifth embodiment of the presentinvention.

FIG. 19 illustrates an example of image synthesis according to the fifthembodiment of the present invention.

FIG. 20 is a block diagram illustrating an exemplary configuration of animage capturing system according to a sixth embodiment of the presentinvention.

FIG. 21 is an illustration of an exemplary application form of the sixthembodiment of the present invention.

FIG. 22 shows graphs conceptually illustrating judgment of whether ornot image synthesis can be performed according to the sixth embodimentof the present invention.

FIG. 23 shows an example of image synthesis according to the sixthembodiment of the present invention.

FIG. 24 is an illustration of another exemplary application form of thesixth embodiment of the present invention.

FIG. 25 is an external view of a cellular phone with cameras equipped toits both sides, respectively.

FIG. 26 is a block diagram illustrating an exemplary configuration of animage capturing system according to a seventh embodiment of the presentinvention.

FIG. 27 shows an example of roll correction according to the seventhembodiment of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

In a first aspect of the present invention, as an image capturingprocessing method, provided is a method including: a first step ofcapturing an image by a camera; a second step of detecting, in an upperregion of the image captured in the first step, feature data of theimage and calculating a correction amount from the feature data; and athird step of performing, based on the correction amount calculated inthe second step, correction control of an optical system of the cameraor correction of image data of the image.

According to a second aspect of the present invention, provided is animage capturing processing method of the first aspect, in which thesecond step includes the steps of: extracting a perpendicular edgeelement in the upper region; detecting, as the feature data, a rollamount of the image based on the perpendicular edge element; andcalculating a roll correction amount of the image based on the rollamount.

According to a third aspect of the present invention, provided is animage capturing processing method of the second aspect, in which in thesecond step, the roll correction amount is calculated using a rollamount at a current time and a roll amount at a previous time.

According to a fourth aspect of the present invention, provided is animage capturing processing method of the second aspect, in which thesecond step includes the steps of: detecting, in addition to the rollamount, a motion vector as the feature data in the upper region; andcalculating a motion correction amount of the image based on the motionvector.

According to a fifth aspect of the present invention, provided is animage capturing processing method of the first aspect, in which thesecond step includes the steps of: obtaining an intensity in the upperregion, detecting an exposure state of the image, as the feature data,based on the obtained intensity; and calculating an exposure correctionamount based on the exposure state.

According to a sixth aspect of the present invention, provided is animage capturing processing method of the fifth aspect, in which theexposure correction is backlight correction or excessive forward lightcorrection.

According to a seventh aspect of the present invention, provided is animage capturing processing method of the first aspect further including:a fourth step of detecting an attitude of the camera; and a fifth stepof adjusting a range of the upper region in the second step according tothe camera attitude detected in the fourth step.

According to an eighth aspect of the present invention, provided is animage capturing processing method of the seventh aspect, in which in thefourth step, data for the detected attitude of the camera is smoothed ina time direction.

According to a ninth aspect of the present invention, as an imagecapturing processing method, provided is a method including: a firststep of capturing images by a plurality of cameras, respectively; asecond step of detecting, in an upper region of each of the imagescaptured in the first step, a roll amount of the image; a third step ofsynthesizing respective roll amounts of the images detected in thesecond step; a fourth step of performing, based on a synthesis rollamount obtained in the third step, roll correction to each of theimages; and a fifth step of synthesizing the images to which rollcorrection has been performed in the fourth step to obtain a synthesizedimage.

According to a tenth aspect of the present invention, as an imagecapturing processing method, provided is a method including: a firststep of capturing images by a plurality of cameras, respectively; asecond step of detecting, in an upper region of each of the imagescaptured in the first step, a roll amount of the image; a third step ofjudging, from respective roll amounts of the images detected in thesecond step, whether to synthesize the images; and a fourth step ofsynthesizing, if it is judged that synthesis should be performed in thethird step, the images to obtain a synthesized image.

According to an eleventh aspect of the present invention, as an imagecapturing processing method, provided is a method including: a firststep of capturing images by first and second cameras, respectively; asecond step of detecting, in an upper region of a first image capturedby the first camera in the first step, a roll amount of the first image;and a third step of performing, based on a roll amount detected in thesecond step, roll correction to a second image captured by the secondcamera.

According to a twelfth aspect of the present invention, as an imagecapturing system, a system including: a camera for capturing an image; aregion limiting section for limiting image data of an image captured bythe camera to an upper region of the image and outputting the limitedimage data; a correction amount calculation section for detecting, fromthe limited image data output from the region limiting section, featuredata of the image and calculating a correction amount from the featuredata; and a correction section for performing, based on a correctionamount calculated by the correction amount calculating section,correction control of an optical system of the camera or correction ofthe image.

Hereinafter, embodiments of the present invention will be described withreference to the accompanying drawings.

FIRST EMBODIMENT

FIG. 1 is a block diagram illustrating the configuration of an imagecapturing system according to a first embodiment of the presentinvention. In FIG. 1, a camera 11 includes a lens and an imaging device.The camera 11 photoelectric-transforms a subject image obtained throughthe lens using an imaging device, thereby generating image data. A rollmechanism 12 rotates an optical system of the camera 11. A roll amountdetection section 13 detects a roll amount of image data output from thecamera 11 with respect to the horizontal direction and then calculates aroll correction amount. A roll control section 14 performs roll controlof the roll mechanism 12, based on the roll correction amount detectedby the roll amount detection section 13.

When the vertical direction of a landscape which has been captured inthe image data does not match the vertical direction in an actual space,the image capturing system of FIG. 1 performs roll correction of theimage data so that the respective vertical directions of the landscapeand the actual space match each other. That is, the image capturingsystem corrects an inclination of the image using a vertical edge. Thisis because, for example, when an image is captured without straightlyfacing an artificial object, a horizontal edge looks inclined in theimage while a vertical edge looks approximately vertical.

The operation of the configuration of FIG. 1 will be described withreference to a flowchart of FIG. 2.

First, the camera 11 transforms a subject into image data (S11). In thiscase, as shown in FIG. 3( a), a window WD of a building as a subject iscaptured from the oblique direction without straightly facing the windowWI). In the image data of FIG. 3( a), each of horizontal and verticaledges of the window WD is inclined. The inclination of the horizontaledge is due not to straightly facing the object. The inclination of thevertical edge is due to roll of the camera 11 on the optical axisdirection.

Next, the roll amount detection section 13 extracts an edge close to thevertical direction and detects a roll amount so as to adjust thedirection of the edge to the perpendicular direction. And the section 13calculates, from the roll amount, a roll correction amount for actuallyroll-controlling the roll mechanism 12 (S12). FIG. 3( b) is a viewillustrating an edge EG extracted from the image data of FIG. 3( b).

Then, the roll control section 14 performs roll control of the rollmechanism 12, based on the obtained roll correction amount. Thus,properly roll-corrected image data can be obtained as shown in FIG. 3(c).

Hereinafter, a method for obtaining the roll correction amount using avertical edge element will be described.

FIG. 4 is a graph showing image data IMG transformed in the xycoordinate system. In this case, P is a pixel in the image data IMG, Iis the intensity of the image data, a is a unit direction vector showingan edge element of the pixel P, and θ is an angle of the unit directionvector a to the positive direction of the x axis.

The directional derivative of the intensity I in coordinate values (x,y) of the pixel P is expressed by Equation 1, based on respectivederivative values in the horizontal and vertical directions and the unitdirection vector a (cos θ, sin θ).

$\begin{matrix}{\frac{\partial I}{\partial a} = {{\frac{\partial I}{\partial x}\cos \; \theta} + {\frac{\partial I}{\partial y}\sin \; \theta}}} & \left\lbrack {{Equation}\mspace{20mu} 1} \right\rbrack\end{matrix}$

In this case, let an evaluation function J be the sum of directionalderivative with respect to the direction of the unit direction vector ain each pixel. Then, J can be expressed by Equation 2.

$\begin{matrix}{J = {\sum\limits_{\theta \in \Theta}\left( \frac{\partial I}{\partial a} \right)^{2}}} & \left\lbrack {{Equation}\mspace{20mu} 2} \right\rbrack\end{matrix}$

In Equation 2, θεΘ means that a subject to be processed is limited. Inthis embodiment, the subject to be processed is limited to a pixelhaving an edge in the approximately vertical direction. For example, theedge direction is limited within ±45 degrees to vertical. In this case,the edge direction and the derivative direction are perpendicular toeach other and thus the derivative direction is limited within ±45degrees to horizontal. Therefore, the pixel needs to satisfy Equation 3.

$\begin{matrix}{\frac{\partial I}{\partial x} \geq \frac{\partial I}{\partial y}} & \left\lbrack {{Equation}\mspace{20mu} 3} \right\rbrack\end{matrix}$

Then, by calculating an angle at which the evaluation function J ismaximum from a pixel satisfying Equation 3 and Equation 4, the rollamount of the image data can be calculated from an edge in theapproximately vertical direction.

$\begin{matrix}{\frac{J}{\theta} = 0} & \left\lbrack {{Equation}\mspace{20mu} 4} \right\rbrack\end{matrix}$

Moreover, the range limiting the edge direction is not only within ±45degrees, but may be appropriately set at a range suitable for an image,i.e., for example, within ±22.5 degrees.

Next, the roll correction amount is calculated. As the most simplemanner, an amount obtained by sign-reversing the above-described rollamount θ may be used as the roll correction amount. However, to performstable roll correction against a rapid scene change and a periodicalintensity change such as a flicker, it is preferable to impose a limitto time change in the roll correction amount to be adapted. For example,the roll correction amount may be obtained using a roll amount at acurrent time and a roll amount at an earlier time.

Therefore, a roll correction amount candidate φ obtained bysign-reversing the roll amount θ may be calculated per field (16.7msec), and then the roll correction amount may be calculated from a rollcorrection amount candidate φt calculated in a current field t and aroll correction amount Φt−1 calculated in the previous field t−1 in themanner expressed by Equation 5.

Φ_(t)=αφ_(t)+(1−α)×Φ_(t-1)  [Equation 5]

where Φt is a roll correction amount applied to the current field t,Φt−1 is a roll correction amount applied to the previous field t−1 and αis a weight coefficient. It has been empirically confirmed by thepresent inventors that setting α at about 0.1-0.01 makes a change in theroll correction amount follow a rapid scene change with a one- ortwo-second delay.

Note that Equation 5 may be reduced to Equation 6 using an obtained rollcorrection amount candidate φt−1.

Φ_(t)=αφ_(t-1)+(1−α)×Φ_(t-1)  [Equation 6]

By calculating the roll correction amount in this manner, it is possibleto impose a limit to a time change in the roll correction amount to beadapted, so that stable roll correction can be performed against a rapidscene change and a periodic illumination change such as a flicker.Therefore, it is possible to prevent roll correction which makes aviewer feel uncomfortable.

Moreover, by not only searching the maximum value of the evaluationfunction J but also obtaining a two-dimensional frequency distributionof derivative values of intensity values in the x and y axis directions,and a frequency distribution of a ratio expressed by Equation 7, anangle θ which is expressed by Equation 8 and appears at the largestfrequency can be detected as the roll amount.

$\begin{matrix}{\frac{\partial I}{\partial y}/\frac{\partial I}{\partial x}} & \left\lbrack {{Equation}\mspace{20mu} 7} \right\rbrack \\{\theta = {\tan^{- 1}\left( {\frac{\partial I}{\partial y}/\frac{\partial I}{\partial x}} \right)}} & \left\lbrack {{Equation}\mspace{20mu} 8} \right\rbrack\end{matrix}$

Note that in this embodiment, the method for calculating a rollcorrection amount from a vertical edge element has been described.However, a roll correction amount can be obtained using a horizontaledge element, a directional filter and the like. For example, when ascene is limited, as in an image captured on the sea and the like, rollcorrection may be performed so that the horizon comes out horizontally.

Note that a roll amount between different frames may be detected. Framesto be compared may be consecutive frames or frames with a timedifference corresponding to two or more frames. For example, a methodfor detecting the roll amount between frames by detecting an opticalflow may be used. In such a case, however, it is necessary to examine adifference between the vertical direction in the actual world and thevertical direction in an image using some other method.

Note that with a gravity sensor installed in the image capturing system,the roll amount may be detected.

Moreover, as the configuration of FIG. 5, the region limiting section 15for detecting a target region in which a vertical edge element isdetected may be provided. Specifically, the region limiting section 15limits image data of an image captured by the camera 11 to an upperregion of the image and outputs the limited image data. Then, the rollamount detection section 13 as a correction amount calculation sectiondetects a roll amount as feature data of the image from the limitedimage data output from the region limiting section 15 and calculates acorrection amount from the roll amount. The roll control section 14 andthe roll mechanism 12 together form a correction section. Thus, forexample, a region can be limited by a feature of image data. Therefore,a horizontal edge element can be efficiently removed, thus resulting inimprovement of accuracy in detecting the roll amount.

For example, when a background of image data is an indoor scenery, thelower region of the image data is considered to include the floor oftenand not to include many vertical edges. In this case, by inputting onlythe upper region in the image data to the roll amount detection section13 by the region limiting section 15, horizontal edge elements can beefficiently removed, thus allowing improvement of accuracy incalculation of the roll correction amount. Furthermore, limitation ofthe target region for detection of a roll amount can be manually set bya user according to a situation.

In this case, as the most simple example, the upper region means to beapproximately upper half of the image. Moreover, the upper region may bea rectangular or oval region included in the approximately upper half ofthe region. Alternatively, the upper region may be a region of whichbarycentric position is located higher than a threshold (e.g., a centerposition of the entire image) among regions obtained by dividing theimage by a known segmentation method.

As has been described, according to this embodiment, an edge close tothe vertical direction is extracted from image data and a roll of thecamera 11 is controlled so that the extracted edge becomes vertical,thereby allowing correction of an inclination of the image with respectto the horizontal direction.

In a wearable image capturing system, especially, a system in which acamera is worn on the head of a person taking an image, an inclinationof the head of the person as it is becomes an inclination of image dataand the inclined image data is reproduced as it is. According to thisembodiment, however, even when the head of a person taking an image isinclined, advantageous effects can be achieved, i.e., image data inwhich the inclination is corrected can be output.

Note that not only in a wearable system but also in a general imagecapturing system, when a scenery including a vertical edge of anartificial object such as the inside of a room and a building is takenas an image, roll correction can be performed to achieve great practicaleffects.

Moreover, if at least one of the roll amount and the roll correctionamount is output, a system according to this embodiment can be used as agyrosensor. Also, if a motion correction amount detected in a thirdembodiment later described is output with the roll amount and/or theroll correction amount, the system of this embodiment can be used as agyrosensor for the roll axis, pitch axis and yawing axis.

Moreover, by accumulating the roll amount, the roll correction amountand the motion amount, how a camera has moved in the actual world can beanalyzed. As another application, it is also possible to save a historyof a movement of an image capturing camera in capturing a movie and thelike and reproduce the camera movement.

Moreover, in using a camera accompanying an auto drive system or acamera movable on or in the water or in the air, a roll amount to becorrected seems to be larger, compared to image capturing by a human. Inimage capturing by such a camera, the roll correction function accordingto this embodiment is effective.

Note that by adding display means such as a display for displaying atleast one of the roll amount and the roll correction amount to theconfiguration of FIG. 1 or FIG. 5, a user can perform directly rollcorrection of the optical system of the camera 11.

SECOND EMBODIMENT

In the first embodiment, roll correction of an image is realized bycorrection control of an optical system of a camera. In contrast,according to this embodiment, roll correction of an image is realized bycorrection of image data. Calculation of a roll correction amount isperformed in the same manner as in the first embodiment and thereforethe detail description will be omitted herein.

FIG. 6 is a block diagram illustrating the configuration of an imagecapturing system according to this embodiment. In FIG. 6, each memberalso shown in FIG. 1 is identified by the same reference numeral. Theroll correction section 21 performs roll correction of image data outputfrom the camera 11 using a roll correction amount output from the rollamount detection section 13.

The operation of the configuration of FIG. 6 will be described withreference to a flowchart of FIG. 7.

First, the camera 11 converts an imaging subject into image data (S21).Next, the roll amount detection section 13 detects a roll amount of animage with respect to the horizontal direction and calculates a rollcorrection amount (S22). A calculation method in this embodiment issimilar to that in the first embodiment and therefore description willbe omitted.

Next, the roll correction section 21 performs roll correction of theimage data output from the camera 11, based on the roll correctionamount calculated by the roll amount detection section 13 and outputs animage of which an inclination is corrected (S23).

In this case, roll correction of the image data is performed in thefollowing manner. As shown in Equation 9, the image data is multipliedby a roll matrix using the roll correction amount θ obtained in the samemanner as in the first embodiment to rotate the image data.

$\begin{matrix}{\begin{pmatrix}{x\; 2} \\{y\; 2}\end{pmatrix} = {{\begin{pmatrix}{\cos \; \theta} & {\sin \; \theta} \\{{- \sin}\; \theta} & {\cos \; \theta}\end{pmatrix}\begin{pmatrix}{{x\; 1} - {cx}} \\{{y\; 1} - {cy}}\end{pmatrix}} + \begin{pmatrix}{cx} \\{cy}\end{pmatrix}}} & \left\lbrack {{Equation}\mspace{20mu} 9} \right\rbrack\end{matrix}$

wherein (x1, y1) indicates pixel coordinates of the original image dataand (x2, y2) indicates pixel coordinates after roll correction. (cx, cy)indicates coordinates of the center of a roll and can be given byproperly setting a center point such as the center of the image datasuitable for the image as the center of a roll.

As has been described, according to this embodiment, roll correction isperformed to image data, thus, not only the same effects as those of thefirst embodiment can be achieved but also correction of an inclinationof an image captured by a camera having no roll mechanism can beperformed.

Moreover, as shown in FIG. 8, the region limiting section 15 describedin the first embodiment may be provided in the subsequent stage of thecamera 11. The roll correction section 21 forms a correction section.Thus, as in the first embodiment, a region in which feature data isdetected can be limited according to a feature of image data, thusresulting in improvement of accuracy in detecting a roll amount.

Note that if display means such as a display for displaying at least oneof a roll amount and a roll correction amount and a roll mechanism forrotating the optical system of the camera 11 are added to theconfiguration of FIG. 6 or FIG. 8, roll correction of the optical systemof the camera 11 can be performed directly by a user.

THIRD EMBODIMENT

In this embodiment, the function of performing motion correction in theup, down, left or right directions by motion detection is added to theimage roll correction function which has been described in the first andsecond embodiments.

FIGS. 9 and 10 are block diagrams illustrating the configuration of animage capturing system according to this embodiment. In FIG. 9, eachmember also shown in FIG. 1 is identified by the same reference numeral.In FIG. 10, each member also shown in FIG. 6 is identified by the samereference numeral.

First, in FIG. 9, a pan/tilt mechanism 31 pans (in the right or leftdirection) and tilts (in the up or down direction) the optical system ofthe camera 11. A motion detection section 33 detects a motion vectorfrom image data output from the camera 11. A pan/tilt control section 32controls the pan/tilt mechanism 31, based on the motion vector detectedby the motion detection section 33. Note that correction control by theroll mechanism 12 is performed in the same manner as in the firstembodiment.

The motion detection section 33 receives the image data output from thecamera 11 and detects a motion vector in the up-down or left-rightdirection. In this case, the motion vector can be obtained from theimage correspondence relation between fields. Next, the pan/tilt controlsection 32 detects a shake of the camera 11 from the motion vectordetected by the motion detection section 33 and calculates a pan/tiltcorrection amount by which the optical system of the camera 11 is pannedor tilted. Then, the pan/tilt mechanism 31 pans or tilts the opticalsystem of the camera 11, based on the pan/tilt correction amount.

Moreover, in FIG. 10, the motion detection section 34 detects a motionvector in the up-down or left-right direction from image dataroll-corrected and output from the roll correction section 21. Themotion correction section 35 corrects a movement of the image data inthe up-down or left-right direction, based on the motion vector outputfrom the motion detection section 34. Note that correction of the imagedata by the roll correction section 21 is performed in the same manneras in the second embodiment.

The motion detection section 34 detects a motion vector in the up-downor left-right direction from image data output from the roll correctionsection 21. In this case, for example, as in the known techniques, themotion vector can be obtained from the image correspondence relationbetween fields. Next, the motion correction section 35 detects a shakeof the image data from the received motion vector, calculates a pan/tiltcorrection amount by which the image data is panned or tilted, andcorrects the image data, based on the pan/tilt correction amount.

As has been described, according to this embodiment, it is possible tocorrect not only an inclination of an image but also a shake in theup-down or left-right direction.

Note that the motion detection section 34 and the motion correctionsection 35 of FIG. 10 may be provided in the subsequent stage of theconfiguration of FIG. 1 of the first embodiment to correct a shake ofthe image data in the up-down or left-right direction.

Note that by adding display means such as a display for displaying atleast one of a roll amount and a roll correction amount and a rollmechanism for rotating the optical system of the camera 11 to theconfiguration of FIG. 9 or FIG. 10, roll correction of the opticalsystem of the camera 11 can be performed directly by a user.

MODIFIED EXAMPLE OF THIRD EMBODIMENT

The configurations of FIGS. 9 and 10 are for detecting a roll amount anda motion vector from an image and correcting a roll and a shake of theoptical system of the camera 11. As a modified example of theconfigurations, it is feasible to achieve a configuration in which amotion amount and a roll amount are from a captured image and, insteadof performing correction on real time, correction is performed when theimage is being stored.

FIG. 11 is a block diagram illustrating an exemplary configurationaccording to the above-described modified embodiment. In FIG. 11, thereference numeral 41 denotes a camera for capturing a motion picture,the reference numeral 42 denotes a region limiting section forextracting image data in a predetermined region of an image captured bythe camera 41, the reference numeral 43 denotes a vertical edgeextraction section for extracting a vertical edge from the imageextracted by the region liming section 42, the reference numeral 44denotes a roll amount detection section for detecting a roll amount ofthe camera 41 from the vertical edge extracted by the vertical edgeextraction section 43, the reference numeral 45 denotes a motion vectordetection section for detecting a motion vector from the image extractedby the region limiting section 42, the reference numeral 46 denotes amotion amount detection section for detecting a motion amount of thecamera 41 from the motion vector detected by the motion vector detectionsection 45, the reference numeral 47 denotes a captured-image correctionsection for correcting a captured image from the motion amount detectedby the roll amount detected by the roll amount detection section 44 andthe motion vector detection section 45, and the reference numeral 48denotes an image output section for outputting the image corrected bythe captured-image correction section 47. Note that the configuration ofFIG. 11 and its operation are the same as those of the embodiments andtherefore the detail description will be omitted.

FIG. 12 is an illustration of an exemplary application form of thismodified example. In FIG. 12, a user wears the camera 41 on the head andimage data captured by the camera 41 is stored in a hard disk recorder49. Then, elements 42 through 48 in the configuration of FIG. 11 areachieved by, for example, a CPU, a memory and the like in the hard diskrecorder 49. A roll amount and a motion amount are detected andcorrection of an image is performed when the image data is being stored.

As shown in FIG. 12, when the camera 41 is worn on the head, detectionof a roll amount is particularly important because of a movement and aninclination of the head. Therefore, by performing correction of the rollamount, an easily-browsable motion picture can be stored.

FOURTH EMBODIMENT

FIG. 13 is a block diagram illustrating the configuration of an imagecapturing system according to a fourth embodiment. In FIG. 13, thereference numeral 131 denotes a camera, the reference numeral denotes aregion limiting section, the reference numeral 133 denotes a backlightdetection section as a correction amount calculation section, and thereference numeral 134 denotes an iris control section. The iris controlsection 134 and an iris (not shown) in the camera 131 together form acorrection section. The region limiting section 132 limits a region inwhich backlight detection for an image captured by the camera 131 isperformed. In a general image capturing performed outside, illuminationsuch as sunlight comes from above a field of view of a camera.Therefore, a processing region is limited to an upper region of an imageby the region limiting section 132.

In this case, it is preferable that the upper portion of an image is setaccording to any one or a combination of the following conditions:

-   -   a rectangular or oval region including part of approximately        upper half region of an image    -   a region of which barycentric position is located higher than a        threshold (e.g., a center position of the entire image) among        regions obtained by dividing the image by a known segmentation        method    -   a portion of the upper region in which an intensity value is a        predetermined threshold or more    -   a portion of the region in which an intensity value is a        predetermined threshold or more of which barycentric position is        located higher than a threshold (e.g., a center position of the        entire image)

The backlight detection section 133 obtains an intensity in a regionlimited by the region limiting section 132 and detects an exposure stateof the image as feature data from the frequency distribution of theintensity. Then, whether or not the image is in a backlight state isjudged, and if it is judged that the image is in a backlight state, anexposure correction amount is calculated. FIG. 14 shows graphsschematically illustrating illumination frequency distributions in aproper illumination state and in a backlight state, respectively. Forexample, in eight bit quantization, whether or not an image is in abacklight state is judged from whether or not a value for a maximumintensity of an input image is 255 or whether the frequency of theintensity of 255 is a threshold or more or the like.

When a backlight state is detected by the backlight detection section133, the iris control section 134 controls the iris of the camera 131according to a given exposure correction amount to reduce the amount ofincident light.

Moreover, an attitude of the camera 131 may be detected and the range ofthe upper region may be adjusted according to the detected cameraattitude. For example, when an optical axis of the camera 131 isapproximately horizontal, the processing region may be limited to theupper region, and when the optical axis of the camera 131 is directedupward, a high intensity region in the image may be the processingregion. Moreover, the attitude of the camera 131 can be detected, forexample, by means such as a gyrosensor.

FIG. 15 is a block diagram illustrating the configuration of an imagecapturing system which can realize the above-described operation. Eachmember also shown in FIG. 13 is identified by the same reference numeraland therefore the detail description will be omitted herein.

An attitude detection section 151 includes, for example, an accelerationsensor such as a gyro, and detects the attitude of the camera 131. AnLPF section 161 smoothes information for the attitude of the camera,detected by the attitude detection section 151 in the time direction. Aregion limiting section 132A adjusts a limited region in the imageaccording to an attitude detection result by the attitude detectionsection 151.

Note that with the LPF section 161 provided, it is possible to preventan abrupt change in a final processing result for the image in the timedirection in the case where the attitude of the camera 131 is rapidlychanged or repeatedly changed with short intervals. Specifically, as theattitude of the camera 131 is rapidly changed or repeatedly changed withshort intervals, a region limited by the region limiting section 132A isdrastically changed. As a result, the final processing result for theimage is drastically changed in the time direction, so that discomfortand unpleasant feelings might be given to a viewer. To avoid this, alimitation can be imposed to a change in the information for thedetected attitude of the camera in the time direction. Note that the LPFsection 161 may be omitted.

As has been described, according to this embodiment, by limiting atarget region in which backlight detection is performed is limited tothe upper region of an image, stable backlight correction can bepreformed.

Note that in this embodiment, as exposure correction, backlightcorrection is performed. However, exposure correction is not limited tobacklight correction. In some other exposure correction such asexcessive forward light correction, when an image is captured under theillumination from above, in general, stable correction can be performedby limiting a detection target region to an upper region of the image.

Moreover, instead of correction control of an optical system of acamera, correction such as intensity correction may be performed toobtained image data.

Moreover, in the case of the above-described roll correction, theattitude of an camera may be detected and the range of an upper regionmay be adjusted according to the detected attitude of the camera.

FIFTH EMBODIMENT

FIG. 16 is a block diagram illustrating an exemplary configuration of animage capturing system according to a fifth embodiment of the presentinvention, in which an image is obtained by a plurality of imagingmeans. In FIG. 16, the reference numerals 51 a and 51 b denote first andsecond cameras for capturing a motion picture, the reference numerals 52a and 52 b denote first and second region limiting sections forextracting image information in a predetermined region of an imagecaptured by the first camera 51 a or the second camera 51 b, thereference numerals 53 a and 53 b denote first and second vertical edgeextraction sections for extracting a vertical edge from the imageextracted from the first region limiting section 52 a or the secondlimiting section 52 b, the reference numerals 54 a and 54 b denote firstand second roll amount detection sections for detecting a roll amount ofthe first camera 51 a or the second camera 51 b from the vertical edgeextracted by the first vertical edge extraction section 53 a or thesecond vertical edge extraction section 53 b, the reference numeral 55denotes a roll amount synthesizing section for synthesizing roll amountsdetected by the first and second roll amount detection sections 54 a and54 b, the reference numerals 56 a and 56 b denote first and secondcaptured-image correction sections for calculating a roll amount of thefirst and second cameras 51 a or 51 b from a roll amount synthesized bythe roll amount synthesizing section 55 and correcting each capturedimage, and the reference numeral 57 denotes an image synthesizingsection for synthesizing images corrected by the first and secondcaptured-image correction sections 56 a and 56 b.

FIG. 17 is an illustration of an exemplary application form of thisembodiment. As shown in FIGS. 17( a) and 17(b), a user wears a firstcamera 51 a (camera 1) and a second camera 51 b (camera 2) on the leftand right of the head, respectively, and image data captured by thecameras 1 and 2 is stored in a hard disk recorder 59. Then, when theimage data is stored, a roll amount is detected, for example, by a CPU,a memory and the like in the hard disk recorder 59 and correction of animage is performed.

As shown in FIG. 17( c), the user can take an image of the front of theuser using the cameras 1 and 2 which the user wears on the left and theright of the head. Respective viewing angles of the cameras 1 and 2overlap each other, and thus captured images have common part. Each ofthe cameras 1 and 2 is worn on the head and therefore physical rollamounts of the cameras are substantially the same. However, what comesout in each of the captured images is different, so that the respectiveroll correction amounts of the cameras might be different. Then, in thisembodiment, a roll angle of the head is detected from the respectiveroll correction amounts of the images obtained from the cameras 1 and 2and correction of each of the images is performed using the detectedroll angle of the head.

FIG. 18 shows graphs conceptually illustrating synthesis of rollcorrection amount according to this embodiment. Each of the graphs ofFIG. 18 shows a distribution of the evaluation function J. FIG. 18( a)shows the distribution thereof for an image of the camera 1, FIG. 18( b)shows the distribution thereof for an image of the camera 2, and FIG.18( c) shows the distribution for after synthesis.

As shown in FIG. 18( a), in the image of the camera 1, an angle at whichthe evaluation function J is a maximal is “−8 degrees” and “+12degrees”. On the other hand. in FIG. 18( b), an angle at which theevaluation function J is a maximal is “−10 degrees”. In this case, forexample, if a roll angle for correcting the image is determined onlyusing the image of the camera 1, the roll angle is “+12 degrees”. Incontrast, according to this embodiment, a roll angle of an image isdetected using both images of the camera 1 and the camera 2. Thus, asshown in FIG. 18( c), the roll angle is determined to be “−9 degrees”.

Accordingly, each of the images captured by the cameras 1 and 2,respectively, is corrected with a roll angle of “−9 degrees”. That is,for cameras worn at the same part, respective roll correction amounts ofthe cameras are synthesized, so that each of the images can be correctedat a proper correction amount.

Then, by correcting the images of the cameras 1 and 2 using a propercorrection amount, the images can be synthesized in the manner as shownin FIG. 19. That is, a roll-corrected, wide-angle image can be obtained.Note that in this case, image synthesis is performed to, for example, aregion in which parts of two images overlap each other, so that adifference between respective pixel values of the images becomes small.

Note that in this embodiment, description has been made using as anexample the case where the roll correction amounts of images aresynthesized. In the same manner, by detecting motion vectors andsynthesizing motion amounts, a proper correction amount forpanning/tilting can be also obtained.

Note that in this embodiment, description has been made using as anexample the case where a plurality of cameras are worn at the left andthe right of the head, respectively. However, the present invention isnot limited thereto. For example, if a plurality of cameras are worn onthe chest or some other part of the human body and relative positions ofthe cameras are fixed, the present invention effectively functions.

Moreover, in this embodiment, a region in which a vertical edge isextracted is limited to the region limiting sections 52 a and 52 b.However, such a region limiting section is not necessary and a verticaledge may be extracted from the entire region of an image.

SIXTH EMBODIMENT

In the fifth embodiment, a more proper roll correction amount isobtained from a roll correction amount obtained from each camera image.Then, after correction has been performed using the roll correctionamount, image synthesis is performed. However, depending on locations onwhich the cameras are worn, there might be cases where images of thecameras should not be synthesized. In this embodiment, whether or notimages captured by the plurality of cameras should be synthesized isjudged based on a roll correction amount.

FIG. 20 is a block diagram illustrating an exemplary configuration of animage capturing system according to this embodiment. In FIG. 20, eachmember also shown in FIG. 16 is identified by the same referencenumeral. The reference numeral 61 denotes an image synthesis judgingsection for judging whether or not images should be synthesized fromroll amounts detected by the roll amount detection sections 54 a and 54b, the reference numerals 62 a and 62 b denote first and secondcaptured-image correction sections for roll-correcting images capturedby the first and second cameras 51 a and 51 b using the roll amountsdetected by the roll amount detection sections 54 a and 54 b, and thereference numeral 63 denotes an image synthesizing section forsynthesizing the corrected images by the first and second captured-imagecorrection sections 62 a and 62 b, respectively, based on a judgmentmade by the image synthesis judging section 61.

FIG. 21 is an illustration of an exemplary application form of thisembodiment. As shown in FIGS. 21( a) and 21(b), a user wears a firstcamera 51 a (camera 1) on the chest and a second camera 51 b (camera 2)on the head. Image data for the front of the cameras captured by thecameras 1 and 2 is stored in the hard disk recorder 59. Then, when theimage data is stored, for example, a roll amount is detected by a CPU, amemory and the like in the hard disk recorder 59 and correction of animage is performed while whether or not image synthesis should beperformed is judged.

In the form of FIG. 17, since both of the cameras 1 and 2 are worn onthe head, the correction amounts of the images are assumed to besubstantially the same. Then, by synthesizing the corrected images, awide-angle image can be obtained. However, as shown in FIG. 21, in thecase where the camera 1 and the camera 2 are worn on different parts,i.e., on the chest and on the head, respectively, for example, if onlythe head turns to the left, the direction in which the camera 1 faces isdifferent from the direction in which the camera 2 faces. In such acase, it is not appropriate to perform image synthesis.

Then, in this embodiment, whether or not image synthesis should beperformed is judged from respective roll correction amounts of thecameras. FIG. 22 shows graphs conceptually illustrating judgment ofwhether or not image synthesis can be performed according to thisembodiment. Each graph of FIG. 22 shows a distribution of theabove-described evaluation function J. In FIG. 22( a), the angle forroll correction of the camera 1 is “−8 degrees” and the angle for rollcorrection of the camera 2 is “−10 degrees”. That is, the angles of thecameras 1 and 2 take relatively close values. In this case, it is judgedthat the cameras 1 and 2 face to the same direction and then imagesynthesis is performed. On the other hand, in FIG. 22( b), the angle forroll correction of the camera 1 is “+11 degrees” and the angle for rollcorrection of the camera 2 is “−12 degrees”. That is, the angles of thecameras 1 and 2 take largely different values. In this case, it isjudged that the cameras 1 and 2 face in the different directions,respectively, and thus image synthesis is not performed. To judge whichthe cameras 1 and 2 face in the same direction or the differentdirections, for example, a difference between roll correction angles canbe compared to an appropriate threshold.

As has been described, whether or not images captured by a plurality ofcameras should be synthesized is judged based on roll correction anglesand, as shown in FIG. 23, if the cameras face in the differentdirections, image synthesis is not performed (FIG. 23( a)) and if thecameras face in the same direction, image synthesis is performed. Thus,a more wide-angle image can be obtained.

Note that in this embodiment, whether or not image synthesis should beperformed is judged using a difference between roll correction angles.However, instead of this, a judgment can be made using, for example, amotion vector. Specifically, a judgment may be made so that a motionvector is detected from each camera image, and if a difference in themotion vector between images is large, synthesis is not performed whileif the difference in the motion vector between images is small, imagesynthesis is performed. Moreover, it is also possible to judge whetheror not image synthesis should be performed using an intensity or aforward light level. Furthermore, whether or not image synthesis shouldbe performed may be judged using respective features of images.

Moreover, in this embodiment, an example where two cameras are used hasbeen described. However, as shown in FIG. 24, three or more cameras maybe used. In the form of FIG. 24, a user wears a camera 1 on the head andcameras 2 and 3 on the left and the right of the head. Moreover, all ofcameras are not necessarily worn the human body. For example, even if acombination of a small movie and a camera worn on the human body isused, this embodiment is applicable.

Moreover, in this embodiment, it is not always required to limit aregion in which a vertical edge is extracted. A vertical edge may beextracted from the entire region of an image.

SEVENTH EMBODIMENT

In the fifth embodiment, a roll correction amount is detected for eachcamera and then detected roll correction amounts are synthesized,thereby obtaining a more proper correction amount. In contrast, in aconfiguration employing a plurality of cameras, it is also possible touse a roll correction amount obtained from an image captured by a camerafor roll correction of some other camera.

Specifically, in recent years, a cellular phone 70 shown in FIG. 25 andhaving a camera 71 a (camera 1) and a camera 71 b (camera 2) equipped toits both sides, respectively, has been put to practical use. As for suchsystems, in many cases, while one of the cameras, i.e., the camera 71 bimages the face of a user himself/herself at a short-distance, the otherof the cameras, i.e., the camera 71 a images an artificial objectlocated at a long distance on the other side. In this case if a rollcorrection amount detected from an image captured by the camera 71 a forperforming long-distance imaging is used for roll correction of thecamera 71 b for performing short-distance imaging, roll correction canbe performed more properly to an image captured by the camera 71 b.

FIG. 26 is a block diagram illustrating an exemplary configuration of animage capturing system according to this embodiment. In FIG. 26, thereference numerals 71 a and 71 b denote first and second cameras forcapturing a motion picture, the reference numeral 72 denotes a regionlimiting section for extracting image information in a predeterminedregion of an image captured by the camera 71 a, the reference numeral 73denotes a vertical edge extraction section for extracting a verticaledge from the image extracted by the region limiting section 72, thereference numeral 74 denotes a roll amount detection section fordetecting a roll amount of the first camera 71 a from the vertical edgeextracted by the vertical edge extraction section 73, and the referencenumeral 75 denotes a captured-image correction section for correcting animage captured by the second camera 71 b using a roll amount detected bythe roll amount detection section 74.

In the cellular phone 70 of FIG. 25, the relative positions of the firstand second cameras 71 a and 71 b are fixed. Thus, when an image capturedby the camera 71 b located on the front surface is inclined, an imagecaptured by the camera 71 a located on the back surface is alsoinclined. Moreover, when camera shake is caused in one of the cameras,camera shake of the other camera is also caused.

Accordingly, as shown in FIG. 27, roll correction of an image of a face,captured by the camera 2 located on the front side is performed (FIGS.27( b) and 27(c)) using the roll correction amount of the image (FIG.27( a)) captured by the camera 1 located on the back side. Specifically,the image captured by the camera 2 is an image in which an object at ashort distance is imaged and an artificial object at a long distance isnot imaged. Thus, there might be cases where roll correction is notproperly performed. However, by making use of a roll correction angledetected from an image captured by the other camera for performinglong-distance imaging, roll correction can be performed more properly.

Note that each means of the image capturing system of each of theembodiments or all or part of process steps of the imaging method ofeach of the embodiments may be achieved by using hardware for exclusiveuse, or may be achieved as software by a computer program. Moreover,assume that all or part of the process steps are achieved by a program.If the program is recorded on a recording medium such as a flexibledisk, an optical disk, an IC card and a ROM cassette and is maderemovable, the all or part of the process steps can be performed in someother independent computer system in a simple manner.

INDUSTRIAL APPLICABILITY

The present invention allows correction of an image captured by a camerain a more proper manner than in a known technique. Accordingly, aclearer image can be obtained in a normal image capturing system or awearable system of which camera is worn at the head and/or the cloth ofa user. Therefore, the present invention is effective.

1. (canceled)
 2. (canceled)
 3. (canceled)
 4. An image capturingprocessing method comprising: a first step of capturing an image by acamera: a second step of detecting, in an upper region of the imagecaptured in the first step, feature data of the image and calculating acorrection amount from the feature data; and a third step of performing,based on the correction amount calculated in the second step, correctioncontrol of an optical system of the camera or correction of image dataof the image, wherein the second step includes the steps of: extractinga perpendicular edge element in the upper region; detecting, as thefeature data, a roll amount of the image based on the perpendicular edgeelement; and calculating a roll correction amount of the image based onthe roll amount, wherein the second step includes the steps of:detecting, in addition to the roll amount, a motion vector as thefeature data in the upper region; and calculating a motion correctionamount of the image based on the motion vector.
 5. An image capturingprocessing method comprising: a first step of capturing an image by acamera; a second step of detecting, in an upper region of the imagecaptured in the first step, feature data of the image and calculating acorrection amount from the feature data; and a third step of performing,based on the correction amount calculated in the second step, correctioncontrol of an optical system of the camera or correction of image dataof the image, wherein the second step includes the steps of: obtainingan intensity in the upper region; detecting an exposure state of theimage, as the feature data, based on the obtained intensity; andcalculating an exposure correction amount based on the exposure state.6. The method of claim 5, wherein the exposure correction is backlightcorrection or excessive forward light correction.
 7. An image capturingprocessing method comprising: a first step of capturing an image by acamera; a second step of detecting, in an upper region of the imagecaptured in the first step, feature data of the image and calculating acorrection amount from the feature data; and a third step of performing,based on the correction amount calculated in the second step, correctioncontrol of an optical system of the camera or correction of image dataof the image. a fourth step of detecting an attitude of the camera; anda fifth step of adjusting a range of the upper region in the second stepaccording to the camera attitude detected in the fourth step.
 8. Themethod of claim 7, wherein in the fourth step, data for the detectedattitude of the camera is smoothed in the time direction.
 9. (canceled)10. (canceled)
 11. (canceled)
 12. (canceled)